<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"  
  "http://www.w3.org/TR/html4/loose.dtd">  
<html > 
<head><title>Joint Chinese Word Segmentation and POS Tagging</title> 
<meta http-equiv="Content-Type" content="text/html; charset=gbk"> 
<meta name="generator" content="TeX4ht (http://www.cse.ohio-state.edu/~gurari/TeX4ht/)"> 
<meta name="originator" content="TeX4ht (http://www.cse.ohio-state.edu/~gurari/TeX4ht/)"> 
<!-- html --> 
<meta name="src" content="joint_seg_tag.tex"> 
<meta name="date" content="2013-03-28 00:53:00"> 
<link rel="stylesheet" type="text/css" href="joint_seg_tag.css"> 
</head><body 
>
   <div class="maketitle">
                                                                          

                                                                          
                                                                          

                                                                          

<h2 class="titleHead">Joint Chinese Word Segmentation and POS
Tagging</h2>
<div class="author" ></div><br />
<div class="date" ><span 
class="ptmr7t-x-x-144">March 28, 2013</span></div>
   </div>
   <h3 class="sectionHead"><span class="titlemark">1    </span> <a 
 id="x1-10001"></a>How to compile</h3>
<!--l. 18--><p class="noindent" >Suppose that ZPar has been downloaded to the directory <span 
class="ptmri7t-x-x-120">zpar</span>. To make the joint
Chinese word segmentor and POS tagger, type <span 
class="ptmri7t-x-x-120">make chinese.postagger</span>. This will create
a directory <span 
class="ptmri7t-x-x-120">zpar/dist/chinese.postagger</span>, in which there are two files: <span 
class="ptmri7t-x-x-120">train </span>and <span 
class="ptmri7t-x-x-120">tagger</span>.
The file <span 
class="ptmri7t-x-x-120">train </span>is used to train a joint model of Chinese word segmentation and POS
tagging,and the file <span 
class="ptmri7t-x-x-120">tagger </span>is used to segment and assign POS tags to new texts using a
trained joint model.
   <h3 class="sectionHead"><span class="titlemark">2    </span> <a 
 id="x1-20002"></a>Format of inputs and outputs</h3>
<!--l. 20--><p class="noindent" >The input files to the <span 
class="ptmri7t-x-x-120">tagger </span>are formatted as a sequence of Chinese characters. An
example input is: <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             ZPar<span 
class="gbksong47-x-x-120">&#x53ef;</span><span 
class="gbksong61-x-x-120">&#x4ee5;</span><span 
class="gbksong41-x-x-120">&#x5206;</span><span 
class="gbksong58-x-x-120">&#x6790;</span><span 
class="gbksong64-x-x-120">&#x4e2d;</span><span 
class="gbksong58-x-x-120">&#x6587;</span><span 
class="gbksong43-x-x-120">&#x548c;</span><span 
class="gbksong62-x-x-120">&#x82f1;</span><span 
class="gbksong58-x-x-120">&#x6587; </span><br 
class="newline" /><br 
class="newline" />The output files contain space-separated words: <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             ZPar_NN <span 
class="gbksong47-x-x-120">&#x53ef;</span><span 
class="gbksong61-x-x-120">&#x4ee5;</span>_VV <span 
class="gbksong41-x-x-120">&#x5206;</span><span 
class="gbksong58-x-x-120">&#x6790;</span>_VV <span 
class="gbksong64-x-x-120">&#x4e2d;</span><span 
class="gbksong58-x-x-120">&#x6587;</span>_NN <span 
class="gbksong43-x-x-120">&#x548c;</span>_CC <span 
class="gbksong62-x-x-120">&#x82f1;</span><span 
class="gbksong58-x-x-120">&#x6587;</span>_NN
<br 
class="newline" /><br 
class="newline" />The output format is also the format of training files for the <span 
class="ptmri7t-x-x-120">train </span>executable.
<br 
class="newline" /><br 
class="newline" />Both input and output files must be encoded in <span 
class="ptmri7t-x-x-120">utf8</span>. Here is a <a 
href="joint_files/gb2utf.py" >script</a> that transfers files
that are encoded in <span 
class="ptmri7t-x-x-120">gb </span>to the <span 
class="ptmri7t-x-x-120">utf8 </span>encoding.
   <h3 class="sectionHead"><span class="titlemark">3    </span> <a 
 id="x1-30003"></a>How to train a model</h3>
<!--l. 37--><p class="noindent" >To train a model, use <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/dist/chinese.postagger/train <span 
class="cmmi-12">&#x003C;</span>train-file<span 
class="cmmi-12">&#x003E; &#x003C;</span>model-file<span 
class="cmmi-12">&#x003E;</span>
<span 
class="cmmi-12">&#x003C;</span>number of iterations<span 
class="cmmi-12">&#x003E; </span><br 
class="newline" /><br 
class="newline" />For example, using the <a 
href="joint_files/train.txt" >example train file</a>, you can train a model by <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                              zpar/dist/chinese.postagger/train train.txt model 1
<br 
class="newline" /><br 
class="newline" />After training is completed, a new file <span 
class="ptmri7t-x-x-120">model </span>will be created in the current directory,
which can be used to do joint segmentation and POS taging to Chinese. The above
command performs training with one iteration (see Section&#x00A0;<a 
href="#x1-60006">6<!--tex4ht:ref: tuning --></a>) using the training
file.
   <h3 class="sectionHead"><span class="titlemark">4    </span> <a 
 id="x1-40004"></a>How to segment and POS-tag new texts</h3>
<!--l. 51--><p class="noindent" >To apply an existing model to do joint segmentation and POS tagging to new texts, use
<br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/dist/chinese.postagger/tagger <span 
class="cmmi-12">&#x003C;</span>model<span 
class="cmmi-12">&#x003E; </span><span 
class="cmr-12">[</span><span 
class="cmmi-12">&#x003C;</span>input-file<span 
class="cmmi-12">&#x003E;</span><span 
class="cmr-12">]</span>
<span 
class="cmr-12">[</span><span 
class="cmmi-12">&#x003C;</span>output-file<span 
class="cmmi-12">&#x003E;</span><span 
class="cmr-12">] </span><br 
class="newline" /><br 
class="newline" />where the input file and output file are optional. If the output file is not specified,
                                                                          

                                                                          
segmented and POS-tagged texts will be printed to the console. If the input file
is not specified, raw texts will be read from the console. For example, using
the model we just trained, we can segment and POS-tag <a 
href="joint_files/input.txt" >an example input</a> by
<br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/dist/chinese.postagger/tagger model input.txt output.txt
<br 
class="newline" /><br 
class="newline" />The output file contains automatically segmented and POS-tagged texts.
   <h3 class="sectionHead"><span class="titlemark">5    </span> <a 
 id="x1-50005"></a>Outputs and evaluation</h3>
<!--l. 65--><p class="noindent" >Automatically segmented and POS-tagged texts contain errors. In order to
evaluate the quality of the outputs, we can manually specify the segmentation
and POS tags of a sample, and compare the outputs with the correct sample.
<br 
class="newline" />A manually specified segmentation and POS tagging of the input file is given in <a 
href="joint_files/reference.txt" >this
example reference file</a>. Here is a <a 
href="joint_files/evaluate.py" >Python script</a> that performs automatic evaluation.
<br 
class="newline" />Using the above <span 
class="ptmri7t-x-x-120">output.txt </span>and <span 
class="ptmri7t-x-x-120">reference.txt</span>, we can evaluate the accuracies by typing
<br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             python evaluate.py output.txt reference.txt <br 
class="newline" /><br 
class="newline" />You can find the precision, recall, and f-score here. See the explanation of these
measures on <a 
href="http://en.wikipedia.org/wiki/Precision_and_recall" >Wikipedia</a>.
   <h3 class="sectionHead"><span class="titlemark">6    </span> <a 
 id="x1-60006"></a>How to tune the performance of a system</h3>
<!--l. 78--><p class="noindent" >The performance of the system after one training iteration may not be optimal. You can
try training a model for another few iterations, after each you compare the performance.
You can choose the model that gives the highest f-score on your test data. We
conventionally call this test file the development test data, because you develop a joint
segmentation and POS tagging model using this. Here is a <a 
href="joint_files/test.sh" >a shell script</a> that
automatically trains the joint segmentor and POS tagger for 30 iterations, and after the
<span 
class="cmmi-12">i</span>th iteration, stores the model file to model.<span 
class="cmmi-12">i</span>. You can compare the f-score of all 30
                                                                          

                                                                          
iterations and choose model.<span 
class="cmmi-12">k</span>, which gives the best f-score, as the final model. In this
file, there is a variable called <span 
class="ptmri7t-x-x-120">zpar</span>. You need to set this variable to the relative directory
of <span 
class="ptmri7t-x-x-120">zpar/dist/chinese.postagger</span>.
   <h3 class="sectionHead"><span class="titlemark">7    </span> <a 
 id="x1-70007"></a>Source code</h3>
<!--l. 80--><p class="noindent" >The source code for the joint segmentor and POS tagger can be found at <br 
class="newline" /><br 
class="newline" />&#x00A0;&#x00A0;&#x00A0;&#x00A0;&#x00A0;                             zpar/src/chinese/tagger/implementation/CHINESE_TAGGER_IMPL
<br 
class="newline" /><br 
class="newline" />where CHINESE_TAGGER_IMPL is a macro defined in <span 
class="ptmri7t-x-x-120">Makefile</span>, and specifies a
specific implementation for the joint segmentor and POS tagger. <br 
class="newline" /><br 
class="newline" />The Chinese POS-tagger by default performs segmentation and tagging simultaneously.
This means that if the input sentence has been segmented, the system will
resegment the sentence. There is one implementation that performs POS-tagging on
segmented sentences. The name of the implementation is <span 
class="ptmri7t-x-x-120">segmented</span>, and you
can compile this system by setting CHINESE_TAGGER_IMPL to <span 
class="ptmri7t-x-x-120">segmented</span>
in <span 
class="ptmri7t-x-x-120">Makefile</span>. The compilation, training, and usage are the same as the other
taggers.
   <h3 class="likesectionHead"><a 
 id="x1-80007"></a>References</h3>
<!--l. 96--><p class="noindent" >
     <div class="thebibliography">
     <p class="bibitem" ><span class="biblabel">
  [1]<span class="bibsp">&#x00A0;&#x00A0;&#x00A0;</span></span><a 
 id="Xbib-1"></a>Yue Zhang and Stephen Clark. 2008. Joint Word Segmentation and POS
     Tagging Using A Single Perceptron. In <span 
class="ptmri7t-x-x-120">Proc. of ACL</span>. pages 888-896.
     </p>
     <p class="bibitem" ><span class="biblabel">
  [2]<span class="bibsp">&#x00A0;&#x00A0;&#x00A0;</span></span><a 
 id="Xbib-2"></a>Yue  Zhang  and  Stephen  Clark.  2010.  A  Fast  Decoder  for  Joint  Word
     Segmentation  and  POS-tagging  Using  a  Single  Discriminative  Model.  In
     <span 
class="ptmri7t-x-x-120">Proc. of EMNLP</span>. pages 843-852.</p></div>
                                                                          

                                                                          
    
</body></html> 

                                                                          


